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ABSTRACT 

K-12 education standards in the U.S. require all students to 
read complex texts across many subject areas. The Language 
Muse™ Activity Palette is a web-based language-instruction 
application that uses NLP algorithms and lexical resources to 
automatically generate language activities and support English 
language learners’ content comprehension and language skills 
development. The system’s online platform for activity gener¬ 
ation, scoring, and feedback is scalable for MOOCs, as well 
as for other online learning settings. 

INTRODUCTION 

The Common Core State Standards adopted by most U.S states 
explicitly emphasize the need for students to read complex sub¬ 
ject area texts to prepare for college and careers [6]. Classroom 
texts may contain language unfamiliar to English language 
learners (ELLs), e.g. figurative language. ELLs could be dis¬ 
advantaged without scaffolding to aid in comprehension of 
unfamiliar language [8]. One way to help is through the use 
of linguistic activities designed to get ELLs familiar with the 
language used in the subject area texts [7]. 

We present the Language Muse Activity Palette (LM hereto¬ 
fore), an open-access, web-based tool 1 , and discuss a small- 
scale instructional pilot intervention that has shown promise 
with regard to addressing ELL content comprehension and lan¬ 
guage learning needs. Teachers can input their own classroom 
texts into LM to automatically generate language activities in 
real-time which can then be assigned to students online. The 
activities are generated using several existing NLP algorithms 
and lexical resources designed to help ELLs with multiple 

1 available at http: //languagemuse. lQclouds. com. 
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aspects of language learning needed to support content com¬ 
prehension: vocabulary, syntactic structures, and discourse 
structure. 

LM is related to existing NLP work on automatic question 
generation [2, 12, 14]. In contrast to previous work, it can 
generate over 20 activity types for any given classroom text, 
covering a large set of language constructs, and offers ac¬ 
tivity customizability. In addition, many activities can be 
automatically scored. Analytics can also be generated for stu¬ 
dents’ language proficiency from both automatically-scored 
and teacher-scored activities. 

Below, we describe the LM NLP backend, the teacher and 
student interactions, findings from a small-scale instructional 
intervention, and future work. 


NLP BACKEND 

LM relies on a backend that uses NLP algorithms to identify 
linguistic features contained in an input text [5]. These fea¬ 
tures include: (a) lexical entities (single word and multi-word 
expressions), (b) syntactic structures, and (c) discourse rela¬ 
tions. LM also relies on a few manually-crafted resources 
either directly, or indirectly as a filter for statistical NLP algo¬ 
rithms that may yield somewhat noisier outputs. This limits 
the teachers’ need to edit over-generated, incorrect options. 

Lexical resources are used for activities related to these lan¬ 
guage elements: homonyms [3], cognates [5], academic words 
[9], and antonyms [10]. Synonym-based activities are pow¬ 
ered using a thresholded combination of WordNet, a distribu¬ 
tional thesaurus [13], and statistically extracted paraphrases 
[1]. Multiword expression activities are generated using a 
rank-ratio based collocation detection algorithm trained on 
the Google Web IT /z-gram corpus [11]. Regular expressions 
defined on constituency parses generate phrasal and sentential 
structures for activities related to contractions, complex verb, 
noun phrases, relative clauses, and multi-clause sentences. A 
morphological analyzer and word-form database are used to 
generate activities related to derivational and inflectional word 
forms. Discourse relations related to cause-effect, compare- 
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Zora Neale Hurston Palette 

Dashboard ► Zora Neale Hurston 


Text Activities (3) 


Text 


✓ 


Click on the text to start editing 


(q) Print text 


Zora Neale Hurston (1891-1960) is perhaps the best known female writer of the Harlem 
Renaissance, even though the novels for which she is famous were published after it. Their Eyes 
Were Watching God (1937) was ranked by TIME magazine as one of the best 100 English Language 
novels published since 1923, when the magazine first began. Still, Hurston was buried in an 
unmarked tomb after suffering a stroke at a Florida welfare home. Thirteen years later, Alice Walker, 
author of The Color Purple, came to the cemetery where she was buried to find and mark a tomb 
believed to be Hurston's in order to properly commemorate her contributions to female African 
American literature. 


Last update: 12/02/2016 2:06 pm 
Description: Add description 

Tags: 

Edit text Delete palette 


While Zora Neale Hurston was bom in Alabama to a Baptist preacher, she was primarily ra... 
Show more... 


Activities 

Sentence activities 

Verb Tense Practice Noun Compounds Discourse Marker Vocabulary 

Created: a few seconds ago Created: a minute ago Created: a minute ago 


-|- Add new activity 


Figure 1. An example activity palette generated using Language Muse. 



Figure 2. A flowchart illustrating the complete Language Muse instruc¬ 
tional workflow from activity generation to students receiving feedback 
on their work. 


contrast, and evidence draw from an adapted rule-based, dis¬ 
course analyzer [4]. 


PALETTE & ASSIGNMENT CREATION, AND SCORING 

Teachers upload a classroom text into LM. The engine au¬ 
tomatically generates over 20 activities based on linguistic 
features identified in the text. Teachers then select activities to 
create an “activity palette” (Figure 1) — a set of text-specific 
activities — to support one or more learning objectives, such 
as “practice with derivational word forms”. 


Full palettes, or specific activities in a palette, can be used 
to create assignments targeting the learning objective. As¬ 
signments can be administered to and completed online by 
students. Multiple choice, and cloze activities are automati¬ 
cally scored. For activities requiring open-ended responses, 
teachers provide scores and written feedback. Teachers and 
students may view scores and feedback at any time. See Fig¬ 
ure 2 for a high-level overview of this workflow. 

INSTRUCTIONAL INTERVENTION 

In Spring 2016, a 6-week instructional intervention study was 
conducted to examine the promise and feasibility of LM use 
in a classroom setting, in preparation for a large-scale random¬ 
ized control trial (RCT) in Spring 2017. 

• Teacher & Student Participants. Results are based on 12 
English Language Arts (ELA), Science, and Social Studies 
teachers from two participating middle schools where ELL 
populations were over 33%. 167 students who completed 
pre- and post-tests were included in the analysis presented 
here. 

• Instruments. The RISE reading assessment [15] was ad¬ 
ministered to students pre- and post-intervention to exam¬ 
ine intervention outcomes. The test contains 6 component 
measures related to reading proficiency, e.g., vocabulary 
and morphology. An observational protocol [5] was used 
to collect teacher observation data pre-, during and post¬ 
intervention. Teacher perception surveys were administered 
post-intervention. 

• Preliminary Findings. The pre-post assessment outcomes 
were difficult to interpret, especially outside of an RCT. 
Gains were observed in some components, but there were 
also score losses - potentially due to motivation, since the 
assessments were no-stakes. Observational data suggested 
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Activity Type 

Count 

Academic Vocabulary 

19 

Antonyms 

9 

Cognates (Spanish) 

17 

Compare/Contrast 

6 

Finding Homonyms 

17 

Multiple Clauses 

16 

Phrasal Verbs 

3 

Referential Terms 

5 

Summary Practice 

3 

Synonyms 

18 

Variant Word Forms 

21 

Verb Tenses 

12 

Word Stems 

9 


Table 1. Teachers’ self-reported usage showing counts of activities dur¬ 
ing the instructional intervention. 


that teachers productively integrated LM into classroom 
instruction. A positive survey finding showed that all teach¬ 
ers successfully completed the intervention, and reported 
that activities completely (47%) or mostly (39%) fulfilled 
intended learning objectives. See Table 1 for activity usage 
during the intervention. 

FUTURE WORK 

In Spring 2017, an instructional intervention will be conducted 
with LM in an RCT with approx. 20 U.S. middle schools with 
high EL populations. Outcomes showing promising use of 
and positive reactions to LM, and pre-post assessment gains 
would suggest promise of LM as a classroom tool, with po¬ 
tential applications in language learning MOOCs and other 
online classrooms to support learning at scale. Currently, Pa¬ 
per Airplanes 2 is also exploring LM use for one-on-one online 
English tutoring for Syrian students. 
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